Why

Two independent faults in backend-tests-integration, both found chasing the CI failure on
run 32238651760
(1 failed, 445 passed, 46 errors). Neither is caused by the PR that run belonged to.

1. A leaked cache override — the 1 failed

FAILED backend/tests/integration/message_bus/operations/request/test_proposed_change.py::TestProposedChange::test_run_generators_validate_requested_jobs
  - infrahub.exceptions.ResourceNotFoundError: Diff summary for pipeline <uuid> was not found in the cache

It is not flaky. It fails exactly when pytest-xdist happens to schedule
backend/tests/integration/proposed_change/test_artifact_regen_e2e.py onto the same worker,
earlier in the session:

memory_cache sets config.OVERRIDE.cache to a MemoryCache and never puts it back. The
dependency_provider.scope(build_cache, ...) unwinds on teardown, but build_cache() consults
config.OVERRIDE.cache first, so the override outlives the class that installed it and every
later get_cache() in that worker returns the dead MemoryCache.

test_run_generators_validate_requested_jobs writes the diff summary through a cache it builds
from config.SETTINGS.cache.driver (Redis) and run_generators reads it back via get_cache().
Once the override leaks, the write goes to Redis and the read goes to the leftover MemoryCache.

bus_simulator, immediately above it, leaks config.OVERRIDE.message_bus the same way. Nothing
fails from it today — a stale BusSimulator silently swallows messages rather than raising —
which is why it is worth closing now. Every other override site in the suite
(component/api/conftest.py, component/telemetry/test_tasks.py, component/git/test_sync_repository.py,
component/api/test_50_internals.py) already saves and restores; these two fixtures were the outliers.

2. A retried Prefect setup — the 46 errors

On gw0 the worker's Prefect testcontainer answered wait_for_prefect and then stopped responding
(httpx.ReadTimeout, dead events websocket) for the rest of the session. The container fault is
infrastructure. The 46 errors are ours: setup_task_manager_once recorded only success, so every
later test class retried the dead server — and since the setup does not fail fast against an
unreachable server, each retry cost the full 300s pytest timeout.

One broken container became 46 Failed: Timeout >300.0s errors across five test files, pushed the
session into its 1800s limit, and buried the original httpx.ReadTimeout under 45 identical copies.

What changed

Three commits, test-only. No production code, so no changelog fragment.

• memory_cache / bus_simulator — save the previous config.OVERRIDE value and restore it in
  a finally, matching the neighbouring workflow_local fixture.
• setup_task_manager_once — remember the failure alongside the success and re-raise it,
  chained, on every later call. The worker still fails, but once, in seconds, with the cause
  attached to the first error rather than the forty-sixth. except BaseException is deliberate:
  the pytest timeout raises Failed, which does not derive from Exception.
• backend/tests/unit/helpers/test_task_manager.py — guards the once-on-success,
  no-retry-on-failure and BaseException cases. The once-per-process state moved onto a
  TaskManagerSetup object taking the setup callable as a constructor argument, so these drive it
  with recording and failing doubles rather than patching the module.

Verification

The cache leak, reproduced by running the two files in one process in the poisoning order:

INFRAHUB_DB_TYPE=neo4j uv run pytest \
  backend/tests/integration/proposed_change/test_artifact_regen_e2e.py \
  backend/tests/integration/message_bus/operations/request/test_proposed_change.py::TestProposedChange::test_run_generators_validate_requested_jobs \
  --neo4j

The retry guard: the three new unit tests pass, and dropping the short-circuit from
setup_task_manager_once fails two of them.

🤖 Generated with Claude Code